Omissions are a major risk in testing. Test planning techniques are a well known way to improve the quality of testing by reducing both omissions and redundancy. These techniques try to prevent omissions in advance (i.e., before tests are implemented and executed) by selecting what to test out of a possibly enormous test space, in a way that reduces as much as possible the risk of bugs escaping to the field. The selection of tests is also characterized in avoiding or reducing tests that are similar to one another and provide no added value. In some cases, three tests may have the same coverage as two tests, as the third test may overlap both two tests (e.g., a first part overlaps the first test and the rest of the test overlaps the second test).
One such known technique is Combinatorial Test Design (CTD). CTD is based on a modeling of the test space. The test space can represent inputs, scenarios, configurations, the application's internal state, or any other aspect that one is interested in testing. The model comprises a set of functional attributes, respective domains of values for each attribute, and restrictions on the value combinations. The restrictions may be Boolean expressions defining when value combinations are valid or invalid. Restrictions may be represented in other manners as well, such as by explicitly enumerating all valid or all restricted combinations of values. The set of valid value combinations, i.e. all combinations from the Cartesian product of the values that pass the restrictions, define the space to be tested.
In CTD, a target level of interaction between sets of attributes may be defined. For example, a level two interaction of all attributes may require that for each two attributes, all combinations of their possible value be checked. A three level interaction would require that for each three attributes, all combinations of their possible values be checked. In some exemplary embodiments, the CTD may define several sets of attributes and interaction levels in them.
Based on the CTD, a description of the test plan may be determined. The description may comprise coverage tasks, each having a value assigned to each functional attribute, and representing a different test to be performed. The description may provide for a coverage as defined (e.g., two level interactions of all attributes). The total number of coverage tasks may be significantly smaller than the total number of possible coverage tasks in accordance with the test space. Some coverage tasks (i.e., tests) may overlap in values of some of their functional attributes.